Image processing apparatus, image reproduction apparatus, system, method and storage medium for image processing and image reproduction

ABSTRACT

An original document image is inputted as multi valued image data (original image data) from an input unit. The multivalued image data is binarized by a binary image generation unit. Then, layout analysis is performed based on the binary image data. Based on the layout information, a partial image having text-attribute is extracted and a partial image having non-text-attribute are extracted from the multi-valued image data. One of the partial images is encrypted, and the encrypted data is stored with the partial image that is not encrypted and the layout information.

CROSS REFERENCE TO RELATED APPLICATION

This is a continuation application U.S. application Ser. No. 09/570,611,filed on May 12, 2000 now U.S. Pat. No. 7,426,054.

FIELD OF THE INVENTION

The present invention relates to an image processing apparatus, imagereproduction apparatus, system, method and storage medium for imageprocessing and image reproduction, and more specifically, to an imageprocessing apparatus, image reproduction apparatus, system, method andstorage medium for reading a document image, performing imageprocessing, and reproducing the document image.

BACKGROUND OF THE INVENTION

Conventional techniques related to document image processing include:copy machines for optically inputting a document image and outputting itby printing the entire image; document database systems for opticallyinputting document data and storing the document data; facsimileapparatuses for optically inputting a document image and outputting thedocument image via a network or communication line; optical characterreaders (OCR) for optically inputting a document image and outputtingtext codes by recognizing characters; and so on.

However, the conventional techniques are no longer applicable todigitized or networked machines. More specifically, because of the factthat a network is employed for connecting an input apparatus with anoutput apparatus and that color documents are handled by theseapparatuses, the following problems arise:

-   (1) The amount of data is too large when an inputted document image    is stored or transmitted without any processing;-   (2) Image quality suitable for reuse cannot be maintained if a    document image is uniformly compressed;-   (3) Quality of an output image may deteriorate depending on whether    the output device is a black-and-white printer or a color printer;-   (4) If texts only are transmitted after performing optical character    recognition (OCR) processing, data such as drawings or photographs    is lost; and-   (5) If erroneous recognition is made by an optical character reader    (OCR), the document may not make sense.

SUMMARY OF THE INVENTION

The present invention has been proposed to solve the above-describedconventional problems, and has as its object to provide an imageprocessing apparatus, image reproduction apparatus, system, method, andstorage medium which can reduce the amount of data of a document imagewhile maintaining a document's layout, and which can suppress imagequality deterioration when reproducing the image.

Furthermore, another object of the present invention is to provide animage processing apparatus, image reproduction apparatus, system,method, and storage medium which can provide a high level of security.

Furthermore, another object of the present invention is to provide animage processing apparatus, image reproduction apparatus, system, methodand storage medium which can accommodate natural language discrepanciesin the texts of an image.

Furthermore, another object of the present invention is to provide animage processing apparatus and method thereof for storing inputteddocument image data with a reduced amount of data, and reproducing thedocument image with high quality by reading the stored data.

In order to solve the above-described problems and achieve the objects,the image processing apparatus according to the present invention hasthe following configuration.

More specifically, the image processing apparatus comprises: input meansfor inputting a document image; analysis means for analyzing an imageattribute of each area and layout of each area, said each areaconstructing the inputted document image; setting means for setting astorage condition of each area based on an analysis result of theanalysis means; and storage means for storing data for each area basedon the storage condition set by the setting means.

Furthermore, according to a preferred embodiment of the presentinvention, the image processing apparatus of the present invention hasthe following configuration.

More specifically, the image processing apparatus comprises: input meansfor inputting an original document as multivalued image data; binaryimage generation means for generating binary image data from theinputted multivalued image data; layout analysis means for analyzing foreach image attribute a layout of a partial image based on the generatedbinary image data, and generating layout data; character recognitionmeans for performing character recognition with respect to the partialimage of a text area based on an analysis result of the layout analysismeans; storage means for adaptively changing a storage condition of eachpartial image based on the analysis result of the layout analysis means,and storing, as image comprehension data, recognition data of the textarea upon which character recognition is performed by the characterrecognition means, image data of an area not subjected to characterrecognition, and layout analysis data obtained by the layout analysismeans; and output means for outputting the image comprehension data,stored by the storage means, to another apparatus.

Furthermore, according to a preferred embodiment of the presentinvention, the image processing apparatus of the present invention hasthe following configuration.

More specifically, the image processing apparatus comprises: image inputmeans for inputting a document image; binary image generation means forgenerating binary image data from the document image inputted by theimage input means; layout analysis means for generating layout analysisdata corresponding to an image attribute included in the document image,based on the binary image data generated by the binary image generationmeans; storage-level setting means for setting a storage level of thedocument image inputted by the image input means; analysis means forgenerating and storing analysis data indicative of a result of analyzingthe document image based on the layout analysis data and the storagelevel set by the storage-level setting means; reproduction-level settingmeans for setting a reproduction parameter for reproducing the documentimage based on the analysis data generated and stored by the analysismeans; and document image reproduction means for reproducing thedocument image in accordance with the analysis data based on thereproduction parameter set by the reproduction-level setting means.

Other features and advantages of the present invention will be apparentfrom the following description taken in conjunction with theaccompanying drawings, in which like reference characters designate thesame or similar parts throughout the figures thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention and,together with the description, serve to explain the principles of theinvention.

FIG. 1 is a block diagram showing data flow according to a firstembodiment of the present invention;

FIG. 2 shows a system construction according to the first embodiment;

FIG. 3 is a flowchart showing the processing steps from inputting animage to outputting document image comprehension data according to thefirst embodiment;

FIG. 4 is a flowchart showing the processing steps from inputtingdocument image comprehension data to outputting a reproduced imageaccording to each embodiment of the present invention;

FIG. 5 is a flowchart showing document image comprehension data storageprocessing according to the first embodiment;

FIG. 6 is a flowchart showing text determination processing according tothe first embodiment;

FIG. 7 is a flowchart showing document image comprehension data storageprocessing according to the first embodiment;

FIG. 8 is a flowchart showing document image comprehension data storageprocessing according to the first embodiment;

FIG. 9 is a flowchart showing document image comprehension datareproduction processing according to the first embodiment;

FIG. 10 is a flowchart showing reproduction image synthesizingprocessing according to the first embodiment;

FIG. 11 is a flowchart showing document image comprehension data storageprocessing according to a second embodiment of the present invention;

FIG. 12 is a flowchart showing document image comprehension data storageprocessing according to a third embodiment of the present invention;

FIG. 13 is a flowchart showing document image comprehension datareproduction processing according to third and fourth embodiments of thepresent invention;

FIG. 14 shows a data structure of document image comprehension dataaccording to the first embodiment;

FIGS. 15A and 15B show a data structure of layout analysis dataaccording to the first embodiment;

FIGS. 16A to 16C show a data structure of image data according to thefirst embodiment;

FIG. 17 shows a data structure of character recognition data accordingto the first embodiment;

FIG. 18 is a block diagram showing data flow according to a fifthembodiment of the present invention;

FIG. 19 is a flowchart showing the processing steps from inputting animage to outputting document image comprehension data according to thefifth embodiment;

FIG. 20 is a flowchart showing document image comprehension data storageprocessing according to the fifth embodiment;

FIG. 21 is a flowchart showing document image comprehension datareproduction processing according to the fifth embodiment;

FIG. 22 is a block diagram showing data flow according to a sixthembodiment of the present invention;

FIG. 23 is a flowchart showing document image comprehension data storageprocessing according to the sixth embodiment;

FIG. 24 is a block diagram showing data flow according to a seventhembodiment of the present invention;

FIG. 25 is a flowchart showing document image comprehension data storageprocessing according to the seventh embodiment;

FIG. 26 is a flowchart showing document image comprehension data storageprocessing according to an eighth embodiment of the present invention;

FIGS. 27A to 27D show a data structure of image data according to thefifth embodiment;

FIG. 28 shows a data structure of character recognition data accordingto the fifth embodiment;

FIG. 29 shows a data structure of encrypted character recognition dataaccording to the fifth embodiment;

FIG. 30 shows an example of graphic user interface (GUI) according tothe sixth embodiment;

FIG. 31 shows an example of graphic user interface (GUI) according tothe seventh embodiment;

FIG. 32 is a block diagram showing data flow according to a ninthembodiment of the present invention;

FIG. 33 is a flowchart showing document image comprehension data storageprocessing according to the ninth embodiment;

FIG. 34 is a flowchart showing document image comprehension datareproduction processing according to the ninth embodiment;

FIG. 35 is a block diagram showing data flow according to a tenthembodiment of the present invention;

FIG. 36 is a flowchart showing document image comprehension data storageprocessing according to an eleventh embodiment of the present invention;

FIG. 37 is a flowchart showing document image comprehension data storageprocessing according to the eleventh embodiment;

FIG. 38 shows an example of graphic user interface (GUI) according tothe tenth embodiment;

FIG. 39 shows an example of graphic user interface (GUI) according tothe eleventh embodiment;

FIG. 40 shows a data structure of document image comprehension dataaccording to the ninth embodiment;

FIG. 41 shows a data structure of translated data according to the ninthembodiment;

FIG. 42 is a block diagram showing a functional configuration of animage processing system according to a thirteenth embodiment of thepresent invention;

FIG. 43 is a flowchart showing the processing steps from inputting animage to outputting document image comprehension data by the imageprocessing system according to the thirteenth embodiment;

FIG. 44 is a flowchart showing document image comprehension data storageprocessing according to the thirteenth embodiment;

FIG. 45 is a flowchart showing document image comprehension data storageprocessing in the storage level 1 according to the thirteenthembodiment;

FIG. 46 is a flowchart showing document image comprehension data storageprocessing in the storage level 2 according to the thirteenthembodiment;

FIG. 47 is a flowchart showing document image comprehension data storageprocessing in the storage level 3 according to the thirteenthembodiment;

FIG. 48 shows a data structure of document image comprehension dataaccording to the thirteenth embodiment;

FIGS. 49A and 49B are tables showing identification numbers for the typeof document images and identification numbers for compression methods;

FIG. 50 is a flowchart showing the processing steps from inputtingdocument image comprehension data to outputting a reproduced imageaccording to the thirteenth embodiment;

FIG. 51 is a flowchart showing document image comprehension datareproduction processing according to the thirteenth embodiment;

FIG. 52 is a flowchart showing document image comprehension datareproduction processing in the reproduction level 1 according to thethirteenth embodiment;

FIG. 53 is a flowchart showing document image comprehension datareproduction processing in the reproduction level 2 according to thethirteenth embodiment;

FIG. 54 is a flowchart showing document image comprehension datareproduction processing in the reproduction level 3 according to thethirteenth embodiment;

FIG. 55 is a flowchart showing reproduction image synthesizingprocessing according to the thirteenth embodiment; and

FIG. 56 is a flowchart showing automatic storage-level settingprocessing according to a fourteenth embodiment of the presentinvention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Preferred embodiments of the present invention will now be described indetail in accordance with the accompanying drawings.

First Embodiment

FIG. 1 is a block diagram showing the structural concept and data flowaccording to the first embodiment.

In FIG. 1, reference numeral 101 denotes an input document includingprinting materials subjected to be inputted or image data processed by acomputer; 102, an image input unit (e.g., image scanner) for inputting adocument as image data; 103, original image data inputted from the imageinput unit 102 (and stored in a predetermined memory); 104, a binaryimage output unit for generating a binary image from the inputteddocument image; 105, generated binary image data; 106, a layoutanalyzing unit for generating and outputting layout analysis data,obtained by dividing the document image into areas having variousattributes, such as drawings, texts, charts or the like, based on thebinary image data 105; and 107, layout analysis data.

Further, reference numeral 108 denotes a character recognitionprocessing unit for recognizing characters in an arbitrary area of thebinary image data based on the layout analysis data and outputting codedata as character recognition data; 109, character recognition data;110, a document image comprehension data storage for storing image dataand character recognition data as document image comprehension data,generated from the inputted original document image and binary imagebased on the layout analysis data while adaptively changing the storagecondition; 111, a document image comprehension data output unit foroutputting document image comprehension data; 112, document imagecomprehension data; 113, a document image comprehension data input unitfor inputting the document image comprehension data; 114, a documentimage comprehension data reproduction unit for generating a reproduceddocument image from the document image comprehension data whileadaptively changing the reproduction condition; 115, a reproduceddocument image; 116, an image output unit for outputting the reproduceddocument image; and 117, an output document.

FIG. 2 shows a system construction according to the first embodiment.

In FIG. 2, reference numeral 201 denotes a computer for performingdocument image comprehension processing (e.g., processing up to thepoint of the document image comprehension data output unit 111 in FIG.1); 202, memory where data is stored; 203, a scanner exclusivelyperforming image input; 204, a color printer; 205, a facsimileapparatus; 206, a digital color copy machine (serving as a color scannerand color printer) for performing a combination of inputting andprinting color images; 207, a printer server for managing printers; 208,a monochrome printer; 209, a file server for managing database; 210,database; 211, a digital copy machine for performing a combination ofinputting and printing monochrome images; and 212, a network whichconnects digital devices.

Next, the processing flow is described with reference to the flowchartsin FIGS. 3 to 10, and FIGS. 1, 2 and 14 to 17.

The first embodiment describes an image comprehension data processingsystem in which an original document image is inputted as 24-bitmultivalued image data (R, G and B each having 8 bits) by a scanner (203or 206), transmitted through a network, and outputted to a monochromeprinter (208, 211, 205) at the transmitted destination.

First in step S301 in FIG. 3, the computer 201 inputs from the imageinput unit 102 (scanner 203 in FIG. 2) a document image as a multivaluedimage.

In step S302, a binary image is generated by the binary image outputunit 104. To obtain binary image data, luminance components arecalculated from the read image data R, G and B (each having 8 bits), andeach of the calculated luminance is compared with a predeterminedthreshold value.

In step S303, based on the binary image generated in the foregoingmanner, the layout analyzing unit 106 divides the document image intoareas having various attributes, such as drawings, texts, charts or thelike, and outputs layout data (FIGS. 16A to 16C). As shown in FIG. 16A,a plurality of rectangular areas (partial images) are set for a page oforiginal document image, and data corresponding to the number of partialimages is stored. Each partial image data consists of identificationinformation (layout area ID), the width of the extracted area, height ofthe extracted area, type of image, compression method, image size, andimage data of the extracted area. The type of image identifies whetherthe partial image is a binary image or 24-bit multivalued image as shownin FIG. 16B. The compression method includes MMR, JPEG or the like asshown in FIG. 16C. Note that the types and number of images andcompression methods are not limited to these mentioned above.

In step S304, character recognition processing is performed on acharacter image in the area including characters such as texts or chartsor the like. In step S305, based on the layout analysis data, documentimage comprehension data is generated from the character recognitiondata, multivalued image, representing the inputted original documentimage, and binary image, while adaptively changing the storagecondition, and the generated data is stored in the document imagecomprehension data storage 110.

Herein, a specific example of document image comprehension data storageprocessing is described with reference to the flowcharts in FIGS. 5 to8.

According to the first embodiment, when the document image comprehensiondata storage processing (step S305 in FIG. 3) is started, the layoutanalysis data of each area is analyzed in step S501 to categorize thearea attribute and document layout type.

Categorizing the document types is described in FIG. 6.

In step S601, an area (AS1) of the entire document is calculated.

In step S602, the sum total (DS1) of areas representing the attributes“texts” and “charts” is calculated. In step S603, it is determinedwhether or not the calculated total area (DS1) accounts for apredetermined ratio (TS1 [%]) with respect to the area (AS1) of theentire document. More specifically, whether or not the followingcondition is satisfied is determined:DS1>AS1×TS1/100If this condition is satisfied, it is determined that the document is a“text-type” layout document (step S604).

Meanwhile in step S603, if it is determined that the document is not the“text-type”, the sum total (PS1) of areas representing the attributes“line drawings” and “pictures and photographs” is calculated (stepS605). In step S606, it is determined whether or not the total area(DS1) calculated for the areas indicative of “texts” and “charts” islarger than the total area (PS1) for the areas indicative of “linedrawings” and “pictures and photographs”. More specifically, it isdetermined if DS1>PSi is satisfied. If YES, it is determined that thedocument is a “text-type.” layout document (step S604).

If NO in step S606, it is determined in step S607 that the document is a“picture-type” layout document.

Referring back to FIG. 5, when the document layout type is determined inthe foregoing manner, the control proceeds to step S502 where thecontrol branches according to the document's layout type. If thedocument's layout type is the “picture-type”, picture-type documentimage comprehension data storage processing is performed in step S503.

The picture-type document image comprehension data storage processing isdescribed in detail with reference to FIG. 7.

When executing the picture-type document image comprehension datastorage processing, layout analysis data of each area is again analyzedin step S701. If the area attribute is “texts” in step S702, a binaryimage is selected in step S703 as the entire image to be extracted, andthe partial image is extracted in step S705 by using coordinate data ofthe area represented by the layout analysis data.

If the area attribute is not “texts” in step S702, a multivalued imageis selected in step S704 as the entire image to be extracted, and themultivalued image, serving as the partial image, is extracted in stepS705 by using coordinate data of the area represented by the layoutanalysis data.

Next, the type of extracted partial image is determined in step S706. Ifit is a binary image type, compression processing for binary images(e.g., MMR or the like) is performed in step S707. The compressedpartial image is stored as document image comprehension data (FIG. 14)in step S709.

At this step, by embedding information indicative of the compressionmethod in the document image comprehension data, the compressed imagecan be decompressed in the document image comprehension datareproduction processing.

Meanwhile, if the extracted partial image is a multivalued image type instep S706, compression processing for multivalued images (e.g., JPEG orthe like) is performed in step S708. The compressed partial image isstored as document image comprehension data (FIG. 14) in step S709.

As exemplified in FIGS. 14 to 16, the document image comprehension datafor an area includes partial image data comprising: an ID uniquelyassigned to the area, the width of the extracted area, height of theextracted area, type of image, compression method employed, image size,and compressed extracted image. Herein, the assigned ID corresponds tothe sequence of area extraction performed by the layout analyzing unit106.

In step S710, it is determined whether or not there is a remaining areato be processed. If there is a remaining area, the control returns tostep S701 for repeating the above-described processing with respect tothe remaining area, whereas if there is no remaining area, the documentimage comprehension data storage processing ends.

Referring back to FIG. 5, in step S502, when it is determined that thedocument's layout type is the “text type”, the control proceeds to stepS504 where text-type document image comprehension data storageprocessing is performed.

The text-type document image comprehension data storage processing isdescribed with reference to the flowchart in FIG. 8, and FIGS. 13 and15.

According to the first embodiment, when the document image comprehensiondata storage processing (step S305 in FIG. 3) is started, the layoutanalysis data of each area is analyzed again in step S801 to obtain anarea attribute.

If the area attribute is “texts” in step S802, character recognitiondata of the area is selected in step S803, and the character recognitiondata is stored as document image comprehension data (FIG. 14) in stepS804.

As exemplified in FIG. 17, the document image comprehension dataaccording to the above example includes character recognition datacomprising: an ID uniquely assigned to the area, the number of lines,line space, 1st to n-th character code candidates for one character,character recognition distance from a standard character to the 1st ton-th candidate characters, type of character font, size of thecharacter, coordinates of the character position, uncertainty flag andso on. Herein, the assigned ID corresponds to the sequence of areaextraction performed by the layout analyzing unit 106.

If the area attribute is not “texts” in step S802, the original image isselected in step S805 as the entire image to be extracted, and thepartial image is extracted in step S806 by using coordinate data of thearea represented by the layout analysis data.

In step S807, since the original image is determined as a multivaluedimage in this example, compression processing for multivalued images(e.g., JPEG or the like) is performed in step S808. The compressedpartial image is stored as document image comprehension data (FIG. 14)in step S809.

At this step, by embedding information indicative of the compressionmethod in the document image comprehension data, the compressed imagecan be decompressed in the document image comprehension datareproduction processing.

As exemplified in FIGS. 16A to 16C, the document image comprehensiondata for an area includes partial image data comprising: an ID uniquelyassigned to the area, the width of the extracted area, height of theextracted area, type of image, compression method employed, image size,and compressed extracted image. Herein, the assigned ID corresponds tothe sequence of area extraction performed by the layout analyzing unit106.

In step S810, it is determined whether or not there is a remaining areato be processed. If there is a remaining area, the control returns tostep S801 for repeating the above-described processing with respect tothe remaining area, whereas if there is no remaining area, the documentimage comprehension data storage processing ends.

Referring back to FIG. 5, when the document image comprehension data foreach layout type is stored in the above-described manner (step S503 orS504), the control proceeds to step S505 in FIG. 5 for outputting thestored document image comprehension data.

According to the first embodiment, the document image comprehensiondata, generated and stored by the document image comprehension datastorage (110), is outputted to a network and transmitted to usersthrough the network.

In the user side (clients), as shown in FIG. 4, document imagecomprehension data is inputted by the document image comprehension datainput unit (113) in step S401. Then, the document image is reproduced instep S402 from the document image comprehension data while adaptivelychanging a reproduction condition.

The document image comprehension data reproduction processing isdescribed in detail with reference to FIG. 9.

In step S901, a white background image is generated to be used as abackground of the reproduced document image.

In step S902, document image comprehension data for one partial area isanalyzed. In step S903, the partial area attribute is inspected, and ifthe attribute is “texts”, it is determined in step S904 whether or notcharacter recognition data exists in the document image comprehensiondata.

If character recognition data exists, the character recognition data(including character codes) is extracted from the document imagecomprehension data in step S905. In step S906, a reproduction image isgenerated by synthesizing the character font with the white backgroundimage based on the extracted character recognition data.

Meanwhile, if the attribute is “non-texts” in step S903 or if there isno character recognition data exists in step S904, image data of thepartial area is extracted from the document image comprehension data instep S907. Based on the extracted partial image data and coordinate datathereof, in step S908, the partial image is synthesized with the whitebackground image, thereby reproducing the image.

An example of reproduction image synthesizing processing (step S908) isdescribed. As shown in FIG. 10, the type of partial image is extractedfrom the document image comprehension data in step S1001. If it isdetermined in step S1002 that the type of image is the “binary imagetype”, pseudo 24-bit conversion is performed in step S1003 byrespectively converting black and white pixels of the binary image toblack and white pixels of a 24-bit multivalued image.

In the first embodiment, assume that a black pixel of the binary imageis expressed by 1, and a white pixel of the binary image is expressed by0. A black pixel of the 24-bit multivalued image is expressed by R=0,G=0, B=0, and a white pixel of the 24-bit multivalued image is expressedby R=255, G=255, B=255 (R: red component; G: green component; B: bluecomponent, each having 8-bit value).

In step S1002, if it is determined that the type of image is the “24-bitmultivalued image type”, the partial image without being processed isused for synthesizing processing.

In step S1004, logical operation is performed on each pixel of thepartial image with respect to the background image and partial image soas to generate a synthesized image.

In the first embodiment, logical operation is performed such that awhite pixel (R=255, G=255, B=255) of the background image, which issynthesized with a black pixel of the partial image (R=0, G=0, B=0),forms a black pixel (R=0, G=0, B=0).

When the reproduction image synthesizing processing (step S908) for onepartial image is completed in the foregoing manner, whether or not thereis a remaining area is determined in step S909. If there is a remainingarea, the control returns to step S902 for repeating the above-describedprocessing with respect to the remaining area, whereas if there is noremaining area, the document image comprehension data reproductionprocessing ends.

Referring back to FIG. 4, after reproduction processing is performed inthe above-described manner, the reproduced image is outputted as anoutput document in step S403 by a monochrome printer serving as theimage output unit 116.

Second Embodiment

In a case where an image condition of a document, having an areaattribute of texts as a result of layout analysis, is inappropriate(existence of noise, low resolution or the like) for characterrecognition processing, it is possible to store certain image datainstead of uncertain character recognition data in the document imagecomprehension data storage processing (step S305 in FIG. 3).

Hereinafter, another specific example of document image comprehensiondata storage processing is described with reference to the flowchart inFIG. 11.

According to the second embodiment, when the document imagecomprehension data storage processing (step S305 in FIG. 3) is started,the layout analysis data of each area is analyzed in step S1101 tocategorize the area attribute and document layout type (FIG. 6).

When it is determined in step S1102 that the document's layout type isthe “picture type”, the control proceeds to step S1106 wherepicture-type document image comprehension data storage processing (FIG.7) is performed.

Meanwhile, when it is determined in step S1102 that the document'slayout type is the “text type”, an overall character recognitionreliability (ZNr) is calculated in step S1103.

An example of calculating the overall character recognition reliabilityis described. In a case where a recognition distance value (D) of thefirst character candidate, which is obtained by recognition calculationperformed on each character, is equal to or larger than a predeterminedthreshold value (Td), that is, in a case whereD≧Tdis satisfied, an uncertainty flag of the character recognition data isvalidated (=1) to indicate that the first character candidate is anuncertain character. Then, the total number (n) of characters having aninvalidated uncertainty flag (=0) is obtained. The ratio (n/N) of theobtained number (n) to the entire number of characters (N) is calculatedas an overall character recognition reliability (ZNr). In other words,the overall character recognition reliability is obtained as follows:ZNr=n/N

Note that a low value of recognition distance (D) indicates that thedistance between the character subjected to character recognition and acharacter recognition candidate is small, i.e., they are similar.

In step S1104, the overall character recognition reliability (ZNr) isinspected. If the overall character recognition reliability is largerthan a predetermined threshold value (Tr), in other words, ifZNr>Tris satisfied, the inspection result is determined to be OK, and thecontrol proceeds to step S1105 where text-type document imagecomprehension data storage processing (FIG. 8) is performed.

If ZNr≦Tr in step S1104, the inspection result is determined to be NG,and the control branches to step S1106 to perform picture-type documentimage comprehension data storage processing (FIG. 7).

When the quality of an original document is low or quality of charactersprinted is low, it is highly likely that erroneous character recognitionis made. According to the above-described second embodiment, in suchcase, the recognition result is not stored, but the document image isstored as a picture-type document. By virtue of this, when the documentimage is reproduced by the client side, high-fidelity reproduction tothe original document image is possible.

Third Embodiment

In the document image comprehension data storage processing (step S305),a particular area of a document, having a “text” attribute as a resultof layout analysis, may be subjected to determination of whether or notthe image condition is appropriate for character recognition (existenceof noise, low resolution or the like). In a case where it is determinedthat the document image is inappropriate for character recognitionprocessing (i.e., character recognition reliability is low), it ispossible to store not only character recognition data, but also imagedata.

Hereinafter, a specific example of document image comprehension datastorage processing is described with reference to the flowchart in FIG.12.

According to the third embodiment, when the document image comprehensiondata storage processing (step S305 in FIG. 3) is started, the layoutanalysis data of each area is analyzed as in FIG. 5 to categorize thearea attribute and document layout type.

When it is determined as a text-type document layout, the layoutanalysis data is analyzed again in step S1201. In step S1202, thepartial area attribute is inspected. If the attribute is “texts”,character recognition data is selected in step S1203, then characterrecognition data storage processing is performed in step S1204, and acharacter recognition reliability (ZMr) of the partial area iscalculated in step S1205.

An example of calculating the partial area character recognitionreliability is described. In a case where a recognition distance value(D), which is obtained by recognition calculation performed on eachcharacter, is equal to or larger than a predetermined threshold value(Td), in other words, in a case whereD≧Tdis satisfied, an uncertainty flag of the character recognition data isvalidated (=1) to indicate that the character candidate is an uncertaincharacter. Then, the total number (m) of characters having aninvalidated uncertainty flag (=0) in the partial area is obtained. Theratio (m/M) of the obtained number (m) to the total number of characters(M) in the partial area is calculated as a partial area characterrecognition reliability (ZMr). In other words, the partial areacharacter recognition reliability is obtained as follows:ZMr=m/M

In step S1206, the partial area character recognition reliability (ZMr)is inspected. If the partial area character recognition reliability islarger than a predetermined threshold value (Tr), more specifically, ifZMr>Tris satisfied, the inspection result is determined to be OK, and theimage data storage processing is not executed. Instead, whether or notthere is a remaining area is determined in step S1214. If there is aremaining area, the control returns to step S1201 for repeating theabove-described processing with respect to the remaining area.

If there is no remaining area, the document image comprehension datareproduction processing ends.

Meanwhile, if ZMR≦Tr is satisfied in step S1206, the inspection resultis determined to be NG, and the image data storage processing (stepsS1207 and S1209-S1213) is performed. In other words, the recognitionresult and image data are both stored. Note that the processing of stepsS1208-S1213 is similar to steps S805-S809.

Fourth Embodiment

In the document image comprehension data reproduction processing, it isalso possible to select either synthesizing character fonts orsynthesizing an image of the area, in accordance with the reliability ofcharacter recognition data.

Hereinafter, this processing is described with reference to theflowchart in FIGS. 4 and 13.

According to the fourth embodiment, the character recognitionreliability is obtained for each partial area, and whether outputprocessing is to be performed with character fonts or with the image ofthe partial area is selected. Assume that area data of a document imageincludes both character recognition data and image data.

In the user side of the document image comprehension data, documentimage comprehension data is inputted by the document image comprehensiondata input unit (113) in step S401 in FIG. 4. Then, the document imageis reproduced in step S402 from the document image comprehension datawhile adaptively changing a reproduction condition.

The document image comprehension data reproduction processing isdescribed in detail with reference to FIG. 13.

In step S1301, a white background image is generated to be used as abackground of the reproduced document image.

In step S1302, document image comprehension data is analyzed. In stepS1303, if the attribute of the partial area is “texts”, it is determinedin step S1304 whether or not character recognition data exists in thedocument image comprehension data. If character recognition data exists,the character recognition data is extracted from the document imagecomprehension data in step S1305, and the character recognitionreliability is determined in step S1306.

To determine the partial area character recognition reliability in thefourth embodiment, the ratio of the uncertainty flag in the characterrecognition data of the partial area is used.

More specifically, the total number (m) of characters having aninvalidated uncertainty flag (=0) in the partial area is obtained. Theratio (m/M) of the obtained number (m) to the total number of characters(M) in the partial area is calculated as a partial area characterrecognition reliability (ZMr). In other words, the partial areacharacter recognition reliability is obtained as follows:ZMr=m/M

As a result of inspecting the partial area character recognitionreliability (ZMr), if the reliability is larger than a predeterminedthreshold value (Tr2), in other words, ifZMr>Tr2is satisfied, then in step S1307, a reproduction image is generated bysynthesizing the character font with the white background image based onthe extracted character recognition data.

Meanwhile, if the partial area character recognition reliability isequal to or less than the predetermined threshold value in step S1306,partial image data is extracted from the document image comprehensiondata in step S1308. Based on the extracted partial image and coordinatedata thereof, in step S1309, the partial image is synthesized with thewhite background image, reproducing the image.

In a case where the attribute of the partial area is “non-texts” in stepS1303 or a case where character recognition data does not exist in stepS1304, partial image data is extracted from the document imagecomprehension data in step S1308, and based on the extracted partialimage and coordinate data thereof, the partial image is synthesized withthe white background image, reproducing the image (step S1309).

After completing the reproduction image synthesizing processing (stepS1309) for one partial image in the foregoing manner, it is determinedin step S1310 whether or not there is a remaining area to be processed.If there is a remaining area, the control returns to step S1302 forrepeating the above-described processing with respect to the remainingarea.

If there is no remaining area, the document image comprehension datareproduction processing ends.

Then in step S403 in FIG. 4, synthesized one page of reproduced image isoutputted as an output document by the image output unit (116).

As has been set forth above, according to the first to fourthembodiments, the following effects are attained:

-   (1) The amount of data is reduced when a document image is stored;-   (2) The load imposed on network traffic is reduced when a document    image is transmitted;-   (3) High quality of a document image suitable for reuse can be    maintained when storing or transmitting the document image;-   (4) Image quality deterioration or data omission can be prevented    when outputting a document image; and-   (5) Electronic use of the document, such as desktop publishing    (DTP), is facilitated.

Fifth Embodiment

In addition to the above-described embodiments, the fifth embodimentattributes importance to security.

FIG. 18 is a block diagram showing the structural concept and data flowaccording to the fifth embodiment.

In FIG. 18, reference numeral 2101 denotes an input document includingprinting materials subjected to be inputted or image data processed by acomputer; 2102, an image input unit for inputting a document as imagedata; 2103, original image data inputted from the image input unit 2102;2104, a binary image output unit for generating a binary image from theinputted document image; 2105, generated binary image data; 2106, alayout analyzing unit for generating and outputting layout analysisdata, obtained by dividing the document image into areas having variousattributes, such as drawings, texts, charts or the like, based on thebinary image data 2105; 2107, layout analysis data; 2108, a documentimage comprehension data storage for storing image data and encrypteddata as document image comprehension data, generated from the inputtedoriginal document image and binary image based on the layout analysisdata while adaptively changing a storage condition; 2109, a documentimage comprehension data output unit for outputting document imagecomprehension data; 2110, document image comprehension data; 2111,document image comprehension data input unit for inputting the documentimage comprehension data; 2112, a document image comprehension datareproduction unit for generating a reproduced document image from thedocument image comprehension data while adaptively changing thereproduction condition; 2113, a reproduced document image; 2114, animage output unit for outputting the reproduced document image; 2115, anoutput document; and 2116, an encryption processing unit characteristicof the fifth embodiment for encrypting character recognition data andimage data of an arbitrary area.

The above-described data flow and processing are realized in a systemsimilar to that shown in FIG. 2.

Next, the processing flow is described with reference to the flowchartsin FIGS. 19 to 21, and FIGS. 18, 2, 14, 15, 27 to 29.

The fifth embodiment describes an image comprehension data processingsystem in which 24-bit multivalued image data is inputted by a scanner(203 or 206), automatic encryption is performed with regard to an areahaving the “text” attribute, then the encrypted data is stored,transmitted through a network, and outputted to a monochrome printer(208, 211, 205) at the transmitted destination.

First in step S2301 in FIG. 19, a document image is inputted asmultivalued image data by a scanner serving as the image input unit(2102 in FIG. 18). In step S2302, a binary image is generated by thebinary image output unit (2104 in FIG. 18). Based on the generatedbinary image, the layout analyzing unit (2106 in FIG. 18) divides thedocument image into various attribute areas, such as drawings, texts,charts or the like, and outputs layout data. The layout data has alreadybeen described in the first embodiment with reference to FIGS. 15A and15B.

In step S2304, based on the layout analysis data, document imagecomprehension data is generated from the multivalued image, representingthe inputted original document image, and binary image, while adaptivelychanging the storage condition, and the generated data is stored in thedocument image comprehension data storage (2108 in FIG. 18). At thisstep, encryption processing for the data in a predetermined specifiedarea is performed.

According to the fifth embodiment, image data in the “text” area isencrypted.

An example of document image comprehension data storage processing isdescribed with reference to the flowchart in FIG. 20, and FIGS. 14, 15and 27.

According to the fifth embodiment, when the document image comprehensiondata storage processing (step S2304 in FIG. 19) is started, the layoutanalysis data of each area is analyzed in step S2501 in FIG. 20 tocategorize the area attribute.

In step S2502, if the area attribute is “texts”, a binary image isselected to be extracted in step S2503. Then in step S2504, the partialimage is extracted by using coordinate data of the area represented bythe layout analysis data.

In step S2505, encryption processing is performed on this partial imageand encrypted data is generated. In step S2506, the encrypted data isstored as document image comprehension data.

As exemplified in FIGS. 27A to 27D, the document image comprehensiondata for an area includes partial image data comprising: an ID uniquelyassigned to the area, the width of the extracted area, height of theextracted area, type of image, compression method employed, image size,and encrypted extracted image. Herein, the assigned ID corresponds tothe sequence of area extraction performed by the layout analyzing unit2106. At this step, validating (=1) an encryption flag indicates thatthe data is encrypted.

If the area attribute is not “texts” in step S2502, a multivalued image,i.e., original image, is selected to be extracted in step S2507. Then instep S2508, the partial image is extracted by using coordinate data ofthe area represented by the layout analysis data.

Next, the type of extracted partial image is determined in step S2509.If it is a binary image type, compression processing for binary images(e.g., MMR or the like) is performed in step S2512. The compressedpartial image is stored as document image comprehension data (FIGS. 27Ato 27D) in step S2511.

At this step, by embedding information indicative of the compressionmethod in the document image comprehension data, the compressed imagecan be decompressed in the document image comprehension datareproduction processing.

Meanwhile, if the extracted partial image is a multivalued image type instep S2509, compression processing for multivalued images (e.g., JPEG orthe like) is performed in step S2510. The compressed partial image isstored as document image comprehension data (FIGS. 27A to 27D) in stepS2511.

As exemplified in FIGS. 27A to 27D, the document image comprehensiondata for an area includes partial image data comprising: an ID uniquelyassigned to the area, the width of the extracted area, height of theextracted area, type of image, compression method employed, image size,and compressed extracted image. Herein, the assigned ID corresponds tothe sequence of area extraction performed by the layout analyzing unit2106.

In step S2513 in FIG. 20, it is determined whether or not there is aremaining area to be processed. If there is a remaining area, thecontrol returns to step S2501 for repeating the above-describedprocessing with respect to the remaining area. If there is no remainingarea, the document image comprehension data storage processing ends.Then, the control proceeds to step S2305 in FIG. 19 to output thedocument image comprehension data.

According to the fifth embodiment, the document image comprehensiondata, generated and stored by the document image comprehension datastorage (2108), is outputted to a network and transmitted to usersthrough the network by the document image comprehension data output unit2109.

In the user side (clients), as described in the first embodiment,processing is performed according to the flowchart shown in FIG. 4. Morespecifically, the document image comprehension data is inputted by thedocument image comprehension data input unit (2111) in step S401, andthe document image is reproduced in step S402 from the document imagecomprehension data while adaptively changing a reproduction condition.

The document image comprehension data reproduction processing (stepS402) according to the fifth embodiment is described with reference tothe flowchart in FIG. 21.

In step S2601, a white background image is generated to be used as abackground of the reproduced document image. In step S2602, documentimage comprehension data is analyzed. In step S2603, if an encryptionflag for encrypted data of the partial area is validated (=1), theencrypted data is extracted from the document image comprehension datain step S2604 to perform decryption processing, and the partial image isrecovered.

If the encryption flag is not validated, image data is extracted fromthe document image comprehension data in step S2605. Then in step S2606,a reproduction image is generated by synthesizing the extracted partialimage with the white background image.

The reproduction image synthesizing processing (step S2606) is performedaccording to the flowchart in FIG. 10 which is described in the firstembodiment.

More specifically, the type of partial image is extracted from thedocument image comprehension data in step S1001. If it is determined instep S1002 that the type of image is the “binary image type”, pseudo24-bit conversion is performed in step S1003 by respectively convertingblack and white pixels of the binary image to black and white pixels ofa 24-bit multivalued image.

In the fifth embodiment, assume that a black pixel of the binary imageis expressed by 1, and a white pixel of the binary image is expressed by0. A black pixel of the 24-bit multivalued image is expressed by R=0,G=0, B=0, and a white pixel of the 24-bit multivalued image is expressedby R=255, G=255, B=255 (R: red component; G: green component; B: bluecomponent, each having 8-bit value).

In step S1002, if it is determined that the type of image is the “24-bitmultivalued image type”, the partial image without being processed isused for synthesizing processing. In step S1004, logical operation isperformed on each pixel of the partial image with respect to thebackground image and partial image so as to generate a synthesizedimage.

In the fifth embodiment, logical operation is performed such that awhite pixel (R=255, G=255, B=255) of the background image, which issynthesized with a black pixel of the partial image (R=0, G=0, B=0),forms a black pixel (R=0, G=0, B=0).

When the reproduction image synthesizing processing (step S2606) for onepartial image is completed in the foregoing manner, whether or not thereis a remaining partial area is determined in step S2607. If there is aremaining area, the control returns to step S2602 for repeating theabove-described processing with respect to the remaining area, whereasif there is no remaining area, the document image comprehension datareproduction processing ends.

Then, the reproduced image is outputted as an output document in stepS403 in FIG. 4 by a monochrome printer serving as the image output unit(2114).

Sixth Embodiment

In the above-described fifth embodiment, a text area is encrypted.According to the sixth embodiment, it is also possible to encrypt aspecified area attribute selected by an operator.

A specific example is described with reference to FIGS. 19, 22 and 23.Note that FIG. 23 is a modification of FIG. 18.

In step S2301 in FIG. 19, a document image (2801) is inputted asmultivalued image data (2803) by a scanner serving as the image inputunit (2802 in FIG. 22).

Next, an area attribute subjected to encryption is designated by anoperator designation unit (2817 in FIG. 22). Herein, for instance, aninputted image is displayed, and designation is made to encrypt an areahaving an attribute “charts.”

In step S2302, a binary image (2805) is generated by the binary imageoutput unit (2804 in FIG. 22). Based on the generated binary image(2805), in step S2303, the layout analyzing unit (2806 in FIG. 22)divides the document image into various attribute areas, such asdrawings, texts, charts or the like, and outputs layout data (2807),(FIGS. 15A and 15B).

In step S2304, based on the layout analysis data (2807) and operator'sdesignation, document image comprehension data is generated from themultivalued image (2803), representing the inputted original documentimage, and binary image (2805), while adaptively changing the storagecondition, and the generated data is stored in the document imagecomprehension data storage (2808 in FIG. 22).

A specific example of document image comprehension data storageprocessing is described with reference to the flowchart in FIG. 23, andFIGS. 14, 15 and 27.

According to the sixth embodiment, when the document image comprehensiondata storage processing (step S2304) is started, an attribute designatedby an operator is set in step S2901 in FIG. 23. FIG. 30 shows a displayscreen at this stage. An operator designates a combination menu box forencryption designation, located in the upper right of the window, andspecifies encryption of a desired area attribute. Assume herein that theoperator designates “texts”. This setting may be performed by pointingat a desired area in the menu with a pointing device or the like.

In step S2902, the layout analysis data of each area is analyzed tocategorize the area attribute. In step S2903, if the area attribute isdesignated to encryption, i.e., if the area attribute is “texts” in thiscase, a binary image is selected in step S2904 as the entire image to beextracted. In step S2905, the partial image is extracted by usingcoordinate data of the area represented by the layout analysis data.

In step S2906, encryption processing is performed on the partial imageand encrypted data is generated. In step S2907, the encrypted data isstored as document image comprehension data.

Validating (=1) an encryption flag indicates that the data is encrypted.

In step S2903, if the area attribute is not designated to encryption,the original image is selected in step S2908 as the entire image to beextracted, and the partial image is extracted in step S2909 by usingcoordinate data of the area represented by the layout analysis data.

Next, the type of extracted partial image is determined in step S2910.If it is a binary image type, compression processing for binary images(e.g., MMR or the like) is performed in step S2911. The compressedpartial image is stored as document image comprehension data (FIGS. 27Ato 27D) in step S2913.

At this step, by embedding information indicative of the compressionmethod in the document image comprehension data, the compressed imagecan be decompressed in the document image comprehension datareproduction processing.

Meanwhile, if the extracted partial image is a multivalued image type instep S2910, compression processing for multivalued images (e.g., JPEG orthe like) is performed in step S2912. The compressed partial image isstored as document image comprehension data (FIGS. 27A to 27D) in stepS2913.

As exemplified in FIGS. 27A to 27D, the document image comprehensiondata for an area includes partial image data comprising: an ID uniquelyassigned to the area, the width of the extracted area, height of theextracted area, type of image, compression method employed, image size,and compressed extracted image. Herein, the assigned ID corresponds tothe sequence of area extraction performed by the layout analyzing unit2806.

In step S2914, it is determined whether or not there is a remaining areato be processed. If there is a remaining area, the control returns tostep S2902 for repeating the above-described processing with respect tothe remaining area, whereas if there is no remaining area, the documentimage comprehension data storage processing ends. Then, the documentimage comprehension data is outputted in step S2305 in FIG. 19.

Seventh Embodiment

In the above-described sixth embodiment, encryption/no encryption isdesignated for each area attribute. According to the seventh embodiment,an operator can confirm an inputted image and layout analysis results,and as a result of analyzing them, the operator can designate encryptionof a desired area by specifying the area.

A specific example is described with reference to FIGS. 19, 24 and 25.

In step S2301 in FIG. 19, a document image is inputted as multivaluedimage data by a scanner serving as the image input unit (3002 in FIG.24).

In step S2302, a binary image is generated by the binary image outputunit (3004 in FIG. 24). Based on the generated binary image, in stepS2303, the layout analyzing unit (3006 in FIG. 24) divides the documentimage into various attribute areas, such as drawings, texts, charts orthe like, and outputs layout data (FIGS. 15A and 15B).

The layout data is superimposed with the inputted image and displayed bythe layout analysis data display unit (3016 in FIG. 24). By this, anoperator is able to designate an area to be encrypted from an operatordesignation unit (3017 in FIG. 24) with a pointing device or the like.FIG. 31 shows an operation screen at this stage. First, an operatordesignates a desired area, and then specifies the setting of encryptionto “ON” with respect to the designated area. The area designated toencryption (text area in the drawing) is displayed distinguishably fromother areas so as to inform the operator that which area is subjected toencryption.

In step S2304, based on the layout analysis data and operator'sdesignation, document image comprehension data is generated from themultivalued image, representing the inputted original document image,and binary image, while adaptively changing the storage condition, andthe generated data is stored in the document image comprehension datastorage (3008 in FIG. 24).

An example of document image comprehension data storage processing isdescribed with reference to the flowchart in FIG. 25, and FIGS. 14, 15and 27.

According to the seventh embodiment, when the document imagecomprehension data storage processing (step S2304) is started, an areaID of the area, designated by an operator, is set in step S3101 in FIG.25. In step S3102, the layout analysis data of each area is analyzed tocategorize the area attribute.

In step S3103, if the area attribute is “texts”, a binary image isselected in step S3104 as the entire image to be extracted. In stepS3106, the partial image is extracted by using coordinate data of thearea represented by the layout analysis data.

In step S3103, if the area attribute is not “texts”, the original imageis selected in step S3105 as the entire image to be extracted, and thepartial image is extracted in step S3106 by using coordinate data of thearea represented by the layout analysis data.

In step S3107, the area ID is inspected to determine whether or not thearea is subjected to encryption.

If the area is subjected to encryption, encryption processing isperformed on the partial image in step S3108 and encrypted data isgenerated. In step S3109, the encrypted data is stored as document imagecomprehension data.

Validating (=1) an encryption flag indicates that the data is encrypted.

If the area is not subjected to encryption in step S3107, the type ofextracted partial image is determined in step S3110. If it is a binaryimage type, compression processing for binary images (e.g., MMR or thelike) is performed in step S3111. The compressed partial image is storedas document image comprehension data (FIGS. 27A to 27D) in step S3113.

At this step, by embedding information indicative of the compressionmethod in the document image comprehension data, the compressed imagecan be decompressed in the document image comprehension datareproduction processing.

Meanwhile, if the extracted partial image is a multivalued image type(non-binary image, e.g., photograph image or the like) in step S3110,compression processing for multivalued images (e.g., JPEG or the like)is performed in step S3112. The compressed partial image is stored asdocument image comprehension data (FIGS. 27A to 27D) in step S3113.

As exemplified in FIGS. 27A to 27D, the document image comprehensiondata for an area includes partial image data comprising: an ID uniquelyassigned to the area, the width of the extracted area, height of theextracted area, type of image, compression method employed, image size,and compressed extracted image. Herein, the assigned ID corresponds tothe sequence of area extraction performed by the layout analyzing unit.

In step S3114, it is determined whether or not there is a remaining areato be processed. If there is a remaining area, the control returns tostep S3102 for repeating the above-described processing with respect tothe remaining area.

If there is no remaining area, the document image comprehension datastorage processing ends. Then, the document image comprehension data isoutputted in step S2305 in FIG. 19.

Eighth Embodiment

With respect to an area of a document image having acharacter-recognizable attribute, e.g., texts, character recognitiondata instead of image data may be stored, or both image data andcharacter recognition data may be stored by using character recognitionprocessing. In such case, it is possible to encrypt the characterrecognition data and image data and store the encrypted data.

Hereinafter, a specific example of this case is described as the eighthembodiment. With reference to FIGS. 18, 19, 26, 14, 15, and 27 to 29,descriptions are provided on an example of storing encrypted characterrecognition data for a “text” area, and storing image data for otherareas.

In step S2301 in FIG. 19, a document image is inputted as multivaluedimage data by a scanner serving as the image input unit (2102 in FIG.18).

In step S2302, a binary image is generated by the binary image outputunit (2104 in FIG. 18). Based on the generated binary image, in stepS2303, the layout analyzing unit (2106 in FIG. 18) divides the documentimage into various attribute areas, such as drawings, texts, charts orthe like, and outputs layout data (FIGS. 15A and 15B).

In step S2304, based on the layout analysis data, document imagecomprehension data is generated from the multivalued image, representingthe inputted original document image, and binary image, while adaptivelychanging the storage condition, and the generated data is stored in thedocument image comprehension data storage (2108 in FIG. 18).

At this step, encryption processing is also performed on the data of apredetermined specific area.

An example of document image comprehension data storage processing isdescribed with reference to the flowchart in FIG. 26, and FIGS. 14, 15and 27 to 29.

According to the eighth embodiment, when the document imagecomprehension data storage processing (step S2304) is started, thelayout analysis data of each area is analyzed to categorize the areaattribute in step S3201 in FIG. 26. In step S3202, if the area attributeis “texts”, a binary image is selected in step S3203 as the image to berecognized. In step S3204, character recognition processing is performedon the partial image by using coordinate data of the area represented bythe layout analysis data.

The character recognition data, obtained by the character recognitionprocessing, is encrypted in step S3205 and encrypted data is generated.In step S3206, the encrypted data is stored as document imagecomprehension data (FIG. 29). Herein, validating (=1) an encryption flagindicates that the data is encrypted.

In step S3202, if the area attribute is not “texts”, the original imageis selected in step S3207 as the entire image to be extracted, and thepartial image is extracted in step S3208 by using coordinate data of thearea represented by the layout analysis data.

The type of extracted partial image is determined in step S3209. If itis a binary image type, compression processing for binary images (e.g.,MMR or the like) is performed in step S3210. The compressed partialimage is stored as document image comprehension data (FIGS. 27A to 27D)in step S3211.

At this step, by embedding information indicative of the compressionmethod in the document image comprehension data, the compressed imagecan be decompressed in the document image comprehension datareproduction processing.

Meanwhile, if the extracted partial image is a multivalued image type instep S3209, compression processing for multivalued images (e.g., JPEG orthe like) is performed in step S3212. The compressed partial image isstored as document image comprehension data (FIGS. 27A to 27D) in stepS3211.

As exemplified in FIGS. 27A to 27D, the document image comprehensiondata for an area includes partial image data comprising: an ID uniquelyassigned to the area, the width of the extracted area, height of theextracted area, type of image, compression method employed, image size,and compressed extracted image. Herein, the assigned ID corresponds tothe sequence of area extraction performed by the layout analyzing unit.

In step S3213, it is determined whether or not there is a remaining areato be processed. If there is a remaining area, the control returns tostep S3201 for repeating the above-described processing with respect tothe remaining area. If there is no remaining area, the document imagecomprehension data storage processing ends. Then, the document imagecomprehension data is outputted in step S2305 in FIG. 19.

Note, further to the above-described processing, the inputtedmultivalued image may be displayed to enable a user to specify an areafor encryption, and with respect to texts in the specified area, boththe text image and character recognition result may be encrypted. Forunauthorized users (who cannot input a decryption code), a message suchas “Text Encrypted. Not Displayable” is presented. When a user points atthe encrypted area with a mouse or the like, a decryption-key-inputdialogue box is displayed to demand decryption key input. When theinputted key coincides with a set number key, the encrypted texts aredisplayed. In the case of performing printing for this area, decryptionprocessing is also required.

As has been set forth above, according to the fifth to eighthembodiments, the following effects are attained:

-   (1) The amount of data is reduced when a document image is stored;-   (2) The load imposed on network traffic is reduced when a document    image is transmitted;-   (3) High quality of a document image suitable for reuse can be    maintained when storing or transmitting the document image;-   (4) Image quality deterioration or data omission can be prevented    when outputting a document image;-   (5) Electronic use of the document, such as desktop publishing    (DTP), is facilitated; and-   (6) Security is improved in storage and transmission of document    images.

Ninth Embodiment

The ninth embodiment includes an additional function of translatingtexts of a document image into another language.

FIG. 32 is a block diagram showing the structural concept and data flowaccording to the ninth embodiment.

In FIG. 32, reference numeral 4101 denotes an input document includingprinting materials subjected to be inputted or image data processed by acomputer; 4102, an image input unit for inputting a document as imagedata; 4103, original image data inputted from the image input unit 4102;4104, a binary image output unit for generating a binary image from theinputted document image; 4105, generated binary image data; 4106, alayout analyzing unit for generating and outputting layout analysisdata, obtained by dividing the document image into areas having variousattributes, such as drawings, texts, charts or the like, based on thebinary image data 4105; 4107, layout analysis data; and 4108, a documentimage comprehension data storage for storing image data, characterrecognition data, and translated data as document image comprehensiondata, generated from the inputted original document image and binaryimage based on the layout analysis data while adaptively changing thestorage condition.

Reference numeral 4109 denotes a document image comprehension dataoutput unit for outputting document image comprehension data; 4110,document image comprehension data; 4111, a document image comprehensiondata input unit for inputting the document image comprehension data;4112, a document image comprehension data reproduction unit forgenerating a reproduced document image from the document imagecomprehension data while adaptively changing the reproduction condition;4113, a reproduced document image; and 4114, an image output unit foroutputting the reproduced document image. Reference numeral 4115 denotesan output document; 4116, a character recognition processing unit forrecognizing characters of an arbitrary area and outputting code data orthe like as character recognition data; and 4117, a translationprocessing unit (including dictionaries or the like for analyzingvarious syntax for translation) for translating data in an arbitrarylanguage into another arbitrary language and outputting the translateddata.

Since the system construction is the same as that shown in FIG. 2,description thereof is omitted.

Next, the processing flow is described with reference to the flowchartsin FIGS. 19, 4, 33 and 10, and FIGS. 32, 2, 40, 15 to 17, and 41.

The ninth embodiment describes an image comprehension data processingsystem in which 24-bit multivalued image data is inputted by a scanner(203 or 206), automatic translation processing is performed with regardto an area having the “text” attribute, then the translated data isstored, transmitted through a network, and outputted to a monochromeprinter (208, 211, 205) at the transmitted destination.

First, in step S2301 in FIG. 19, a document image is inputted asmultivalued image data by a scanner serving as the image input unit(4102 in FIG. 32).

In step S2302, a binary image is generated by the binary image outputunit (4104 in FIG. 32). Based on the generated binary image, in stepS2303, the layout analyzing unit (4106 in FIG. 32) divides the documentimage into various attribute areas, such as drawings, texts, charts orthe like, and outputs layout data (FIGS. 15A and 15B).

In step S2304, based on the layout analysis data, document imagecomprehension data is generated from the multivalued image, representingthe inputted original document image, and binary image, while adaptivelychanging the storage condition, and the generated data is stored in thedocument image comprehension data storage (4108 in FIG. 32). At thisstep, character recognition processing and translation processing arealso performed on the data of a predetermined specific area.

The ninth embodiment describes an example of performing characterrecognition and translation processing (Japanese into English) on imagedata of the “text” area.

An example of document image comprehension data storage processing isdescribed with reference to the flowchart in FIG. 33, and FIGS. 40, 15and 16.

According to the ninth embodiment, when the document image comprehensiondata storage processing (step S2304 in FIG. 19) is started, the layoutanalysis data of each area is analyzed to categorize the area attributein step S4501 In FIG. 33.

In step S4502, if the area attribute is “texts”, a binary image isselected in step S4503 as the image to be recognized. In step S4504,character recognition processing of the partial image is performed byusing coordinate data of the area represented by the layout analysisdata.

The character recognition data, obtained by the character recognitionprocessing, is stored as document image comprehension data (FIGS. 16A to16C) in step S4505.

In step S4506, translation processing is performed on the characterrecognition data, obtained as a result of character recognition, andtranslated data is generated. In step S4507, the translated data is alsostored as document image comprehension data (FIG. 41).

In step S4502, if the area attribute is not “texts”, the original imageis selected in step S4508 as the entire image to be extracted, and thepartial image is extracted in step S4509 by using coordinate data of thearea represented by the layout analysis data.

The type of extracted partial image is determined in step S4510. If itis a binary image type, compression processing for binary images (e.g.,MMR or the like) is performed in step S4511. The compressed partialimage is stored as document image comprehension data (FIGS. 16A to 16C)in step S4513.

At this step, by embedding information indicative of the compressionmethod in the document image comprehension data, the compressed imagecan be decompressed in the document image comprehension datareproduction processing.

Meanwhile, if the extracted partial image is a multivalued image type instep S4510, compression processing for multivalued images (e.g., JPEG orthe like) is performed in step S4512. The compressed partial image isstored as document image comprehension data (FIGS. 16A to 16C) in stepS4513.

As exemplified in FIGS. 16A to 16C, the document image comprehensiondata for an area includes partial image data comprising: an ID uniquelyassigned to the area, the width of the extracted area, height of theextracted area, type of image, compression method employed, image size,and compressed extracted image. Herein, the assigned ID corresponds tothe sequence of area extraction performed by the layout analyzing unit.

In step S4514, it is determined whether or not there is a remaining areato be processed. If there is a remaining area, the control returns tostep S4501 for repeating the above-described processing with respect tothe remaining area. If there is no remaining area, the document imagecomprehension data storage processing ends. Then, the document imagecomprehension data is outputted in step S2305 in FIG. 19.

According to the ninth embodiment, the document image comprehensiondata, generated and stored by the document image comprehension datastorage (4108), is outputted to a network and transmitted to usersthrough the network by the document image comprehension data output unit(4109).

In the user side (clients), as described in the first embodiment,processing is performed according to the flowchart shown in FIG. 4.

More specifically, the document image comprehension data is inputted bythe document image comprehension data input unit (4111) in step S401,and the document image is reproduced in step S402 from the documentimage comprehension data while adaptively changing a reproductioncondition.

The document image comprehension data reproduction processing (stepS402) according to the ninth embodiment is described with reference toFIGS. 34 and 10.

The ninth embodiment describes an example where document imagecomprehension data includes translated data of a predetermined language,and where the translated data is automatically extracted and reproduced.

In step S4601 in FIG. 34, a white background image is generated to beused as a background of the reproduced document image.

In step S4602, document image comprehension data is analyzed. In stepS4603, if the attribute of the partial area is “texts”, it is determinedin step S4604 whether or not document image comprehension data includestranslated data.

If translated data is found in step S4604, the translated data isextracted from the document image comprehension data in step S4605. Thenin step S4606, a font pattern corresponding to a character codeconstituting the translated sentence is synthesized with the whitebackground image, reproducing the image.

If translated data is not found in step S4604, character recognitiondata of the original text is extracted from the document imagecomprehension data in step S4607.

In step S4608, the extracted original text is synthesized with the whitebackground image, generating a reproduced image.

In step S4603, in a case where the attribute of the partial area is not“texts”, image data is extracted from the document image comprehensiondata in step S4609. Then in step S4610, based on the extracted partialimage and coordinate data thereof, the partial image is synthesized withthe white background image, reproducing the image.

The reproduction image synthesizing processing (step S4610) is performedaccording to the flowchart in FIG. 10, which is described in the firstembodiment. More specifically, the type of partial image is extractedfrom the document image comprehension data in step S1001. If it isdetermined in step S1002 that the type of image is the “binary imagetype”, pseudo 24-bit conversion is performed in step S1003 byrespectively converting black and white pixels of the binary image toblack and white pixels of a 24-bit multivalued image.

In the ninth embodiment, assume that a black pixel of the binary imageis expressed by 1, and a white pixel of the binary image is expressed by0. A black pixel of the 24-bit multivalued image is expressed by R=0,G=0, B=0, and a white pixel of the 24-bit multivalued image is expressedby R=255, G=255, B=255 (R: red component; G: green component; B: bluecomponent, each having 8-bit value).

In step S1002, if it is determined that the type of image is the “24-bitmultivalued image type”, the partial image without being processed isused for synthesizing processing.

In step S1004, logical operation is performed on each pixel of thepartial image with respect to the background image and partial image soas to generate a synthesized image.

When the reproduction image synthesizing processing (step S4606, S4608,S4610) for one partial image is completed in the foregoing manner,whether or not there is a remaining partial area is determined in stepS4611. If there is a remaining area, the control returns to step S4602for repeating the above-described processing with respect to theremaining area, whereas if there is no remaining area, the documentimage comprehension data reproduction processing ends.

Then, the reproduced image is outputted as an output document in stepS403 in FIG. 4 by a monochrome printer serving as the image output unit(4114).

Tenth Embodiment

According to the tenth embodiment, an operator can confirm an inputtedimage and layout analysis results, and as a result of analyzing them,the operator can designate translation of a desired area by specifyingthe area.

A specific example is described with reference to FIGS. 19 (described inthe fifth embodiment), 35 and 36.

In step S2301 in FIG. 19, a document image is inputted as multivaluedimage data by a scanner serving as the image input unit (4802 in FIG.35).

In step S2302, a binary image is generated by the binary image outputunit (4804 in FIG. 35). Based on the generated binary image, in stepS2303, the layout analyzing unit (4806 in FIG. 35) divides the documentimage into various attribute areas, such as drawings, texts, charts orthe like, and outputs layout data (FIGS. 15A and 15B).

The layout data is superimposed with the inputted image and displayed bythe layout analysis data display unit (4816 in FIG. 35). By this, anoperator is able to designate an area to be translated from an operatordesignation unit (4817 in FIG. 35) with a pointing device or the like.FIG. 38 shows an operation screen for designating an area to betranslated. In FIG. 38, the upper part of the second column is set as anarea subjected to translation (translation area designation is “ON”).

In step S2304, based on the layout analysis data and operator'sdesignation, document image comprehension data is generated from themultivalued image, representing the inputted original document image,and binary image, while adaptively changing the storage condition, andthe generated data is stored in the document image comprehension datastorage (4808 in FIG. 35).

An example of document image comprehension data storage processing isdescribed with reference to the flowchart in FIG. 36, and FIGS. 40, 15and 16.

According to the tenth embodiment, when the document image comprehensiondata storage processing (step S2304) is started, an area ID of the area,designated by an operator, is set in step S4901 in FIG. 36.

In step S4902, the layout analysis data of each area is analyzed tocategorize the area attribute. In step S4903, if the area attribute is“texts”, a binary image is selected in step S4904 as the image to berecognized. In step S4905, character recognition processing is performedon the partial image by using coordinate data of the area represented bythe layout analysis data. In step S4906, the character recognition datais stored as document image comprehension data. Note with regard toimages not including texts, e.g., photograph images or the like, it isso set that translation is not performed. Thus, problems do not arise insuch areas.

In step S4907, the area ID of the partial area is inspected to determinewhether or not the area is subjected to translation.

If the area is not a translation-subjected area, character recognitiondata of the original text is stored, and the processing for this areaends.

If the area is a translation-subjected area, translation processing isperformed in step S4908. In step S4909, the translated data is stored asdocument image comprehension data.

Meanwhile in step S4903, if the area attribute is not atranslation-subjected category, i.e., if texts are not included, theoriginal image is selected in step S4910 as the entire image to beextracted, and the partial image is extracted in step S4911 by usingcoordinate data of the area represented by the layout analysis data.

In step S4912, the type of extracted partial image is determined. If itis a binary image type, compression processing for binary images (e.g.,MMR or the like) is performed in step S4913. The compressed partialimage is stored as document image comprehension data (FIGS. 16A to 16C)in step S4915.

At this step, by embedding information indicative of the compressionmethod in the document image comprehension data, the compressed imagecan be decompressed in the document image comprehension datareproduction processing.

Meanwhile, if the extracted partial image is a multivalued image type instep S4912, compression processing for multivalued images (e.g., JPEG orthe like) is performed in step S4914. The compressed partial image isstored as document image comprehension data (FIGS. 16A to 16C) in stepS4915.

As exemplified in FIGS. 16A to 16C, the document image comprehensiondata for an area includes partial image data comprising: an ID uniquelyassigned to the area, the width of the extracted area, height of theextracted area, type of image, compression method employed, image size,and compressed extracted image. Herein, the assigned ID corresponds tothe sequence of area extraction performed by the layout analyzing unit.

In step S4916, it is determined whether or not there is a remaining areato be processed. If there is a remaining area, the control returns tostep S4902 for repeating the above-described processing with respect tothe remaining area.

If there is no remaining area, the document image comprehension datastorage processing ends. Then, the document image comprehension data isoutputted in step S2305 in FIG. 19.

Eleventh Embodiment

According to the eleventh embodiment, which can be a variation ofprevious embodiments, such as the eighth and ninth embodiments,translation means for a plurality of languages can be provided so thatcharacter recognition data of an original document can includetranslation data in plural languages.

Twelfth Embodiment

According to the twelfth embodiment, language to be translated can beselected by an operator from a plurality of languages.

A specific example is described with reference to FIGS. 19, 35 and 37.

In step S2301 in FIG. 19, a document image is inputted as multivaluedimage data by a scanner serving as the image input unit (4802 in FIG.35).

Next, a language to be translated is designated from the operatordesignation unit (4817 in FIG. 35).

Assume herein that the operator designates “Japanese” as the language tobe translated from an original English-text document. Although onelanguage is designated in this embodiment, a plurality of languages maybe designated.

In step S2302, a binary image is generated by the binary image outputunit (4804 in FIG. 35). Based on the generated binary image, in stepS2303, the layout analyzing unit (4806 in FIG. 35) divides the documentimage into various attribute areas, such as drawings, texts, charts orthe like, and outputs layout data (FIGS. 15A and 15B).

In step S2304, based on the layout analysis data and operator'sdesignation, document image comprehension data is generated from themultivalued image, representing the inputted original document image,and binary image, while adaptively changing the storage condition, andthe generated data is stored in the document image comprehension datastorage (4808 in FIG. 35).

The operator's designation herein is made by selecting a language to betranslated from a display screen shown in FIG. 39, which shows contentsof the layout analysis. In the twelfth embodiment, although the areasubjected to translation is a text area, other areas includingcharacters, e.g., charts or the like, may be subjected to translation.

An example of document image comprehension data storage processing isdescribed with reference to the flowchart in FIG. 37, and FIGS. 40, 15to 17, and 41.

According to the twelfth embodiment, when the document imagecomprehension data storage processing (step S2304) is started, thetranslation language designated by an operator is set in step S5001 inFIG. 37 to prepare for the translation processing.

In step S5002, the layout analysis data of each area is analyzed tocategorize the area attribute.

In step S5003, it is determined if the area attribute is “texts”. If so,in step S5004, a binary image is selected as the image to be recognized.In step S5005, character recognition processing of the partial image isperformed by using coordinate data of the area represented by the layoutanalysis data. In step S5006, the character recognition data is storedas document image comprehension data. In step S5008, translationprocessing is performed. Then in step S5009, the translated result isstored (FIG. 41).

Meanwhile in step S5003, if the area attribute is not “texts”, theoriginal image is selected in step S5010 as the entire image to beextracted, and the partial image is extracted in step S5011 by usingcoordinate data of the area represented by the layout analysis data.

In step S5012, the type of extracted partial image is determined. If itis a binary image type, compression processing for binary images (e.g.,MMR or the like) is performed in step S5013. The compressed partialimage is stored as document image comprehension data (FIGS. 16A to 16C)in step S5015.

At this step, by embedding information indicative of the compressionmethod in the document image comprehension data, the compressed imagecan be decompressed in the document image comprehension datareproduction processing.

Meanwhile, if the extracted partial image is a multivalued image type instep S5012, compression processing for multivalued images (e.g., JPEG orthe like) is performed in step S5014. The compressed partial image isstored as document image comprehension data (FIGS. 16A to 16C) in stepS5015.

As exemplified in FIGS. 16A 16C, the document image comprehension(non-text image) according to the twelfth embodiment includes partialimage data comprising: an ID uniquely assigned to the area, the width ofthe extracted area, height of the extracted area, type of image,compression method employed, image size, and compressed extracted image.Herein, the assigned ID corresponds to the sequence of area extractionperformed by the layout analyzing unit.

In step S5016, it is determined whether or not there is a remaining areato be processed. If there is a remaining area, the control returns tostep S5002 for repeating the above-described processing with respect tothe remaining area. If there is no remaining area, the document imagecomprehension data storage processing ends. Then, the document imagecomprehension data is outputted in step S2305 in FIG. 19.

As has been set forth above, according to the ninth to twelfthembodiments, the following effects are attained:

-   (1) The amount of data is reduced when a document image is stored;-   (2) The load imposed on network traffic is reduced when a document    image is transmitted;-   (3) High quality of a document image suitable for reuse can be    maintained when storing or transmitting the document image;-   (4) Image quality deterioration or data omission can be prevented    when outputting a document image;-   (5) Electronic use of the document, such as desktop publishing    (DTP), is facilitated; and-   (6) Improved convenience in worldwide transmission of document    images.

Note that although FIG. 2 shows data transmission/reception through anetwork, the Internet may be used as the network. The form of networkdoes not limit the present invention.

Also note that the above-described embodiments can be roughlycategorized into first to fourth embodiments, fifth to eighthembodiments, and ninth to twelfth embodiments. However, each of theseembodiments can be combined in any ways.

As has been set forth above, according to the above embodiments, theamount of data of an original document can be reduced while maintainingthe layout of the original document, and image quality deterioration canbe prevented at the time of reproducing a document image.

Furthermore, improved security is achieved when transmitting documentimages.

Still further, natural language discrepancies in the texts of an imagecan be accommodated. Therefore, information can be shared on a worldwidebasis.

Thirteenth Embodiment

FIG. 42 is a block diagram showing a functional configuration of animage processing system according to the thirteenth embodiment of thepresent invention.

In FIG. 42, reference numeral 5101 denotes an input original documentincluding printing materials or image data processed by a computer;5102, an image input unit including, e.g., image scanner or the like,for reading the original document 5101 and inputting the data as imagesignals; 5103, original image data inputted from the image input unit5102; 5104, a binary image output unit for generating binary image data5105 from the image signals of the inputted document 5101; and 5106, alayout analyzing unit for generating and outputting layout analysis data5107, obtained by dividing the inputted document image into areas havingvarious attributes, such as drawings, texts, charts or the like, basedon the binary image data 5105. Reference numeral 5108 denotes a storagelevel setting unit for analyzing and comprehending the inputted documentimage, and setting a storage level serving as a parameter to obtaindocument image comprehension data 5111; 5109, a document imagecomprehension data storage for storing document image comprehension datagenerated from the inputted original image data 5103 and binary imagedata 5105 while adaptively changing the storage condition; 5110, adocument image comprehension data output unit for outputting documentimage comprehension data 5111 by reading the data stored in the documentimage comprehension data storage 5109; and 5112, a document imagecomprehension data input unit for inputting the document imagecomprehension data 5111 which is outputted from the document imagecomprehension data output unit 5110. The document image comprehensiondata output unit 5110 and document image comprehension data input unit5112 may be connected through a network 5212 which will be describedlater. Reference numeral 5113 denotes a reproduction level setting unitfor setting a parameter to reproduce the original document image basedon the document image comprehension data 5111; 5114, a document imagecomprehension data reproduction unit for generating a reproduceddocument image from the document image comprehension data 5111 whileadaptively changing the reproduction condition; 5115, reproduceddocument image data; 5116, an image output unit for outputting thereproduced document image 5115; and 5117, an output document.

The construction of an image processing system according to thethirteenth embodiment is the same as that shown in FIG. 2. Thus,detailed description thereof is omitted herein.

Next, the processing flow of the image processing system according tothe thirteenth embodiment is described.

FIG. 43 is a flowchart showing the processing steps of the imageprocessing system according to the thirteenth embodiment.

The thirteenth embodiment describes the image processing system in which24-bit multivalued image data is inputted by the scanner 203 or copymachine 206, then transmitted through the network 212, and outputted tothe monochrome printer 208 or copy machine 211 or facsimile apparatus205 at the transmitted destination.

First in step S6301 in FIG. 43, a document image is inputted asmultivalued image data by the scanner 203 serving as the image inputunit 5102. In step S6302, the multivalued image data is converted tobinary image data 5105 by the binary image output unit 5104. In stepS6303, based on the generated binary image data 5105, the layoutanalyzing unit 5106 divides the document image into various attributeareas, such as drawings, texts, charts or the like, and outputs layoutdata 5107 (FIGS. 15A and 15B).

As shown in FIGS. 15A and 15B, layout analysis data 5107 includes thenumber of divided areas n, X and Y coordinates, width, height andattribute (texts=1, line drawings=2, pictures and photographs=3,charts=4) of each area.

Next in step S6304, the storage level is set by the storage levelsetting unit 5108. According to the thirteenth embodiment, an operatorcan set any of “level 1” to “level 3”. The operator's storage-levelsetting is performed by an input operation from a keyboard or touchpanel of the terminal in the system shown in FIG. 2. The storage level,once set, may be stored until the next time the storage level ischanged, and may repeatedly be used for different input images. In stepS6305, based on the layout analysis data 5107 and set storage level,document image comprehension data 5111 is generated from the multivaluedimage data 5103 representing the inputted original document image andbinary image data 5105, while adaptively changing the storage condition,and the generated data is stored in the document image comprehensiondata storage 5109. In step S6306, the document image comprehension data5111 is outputted. At this step, a tag indicative of the storage level,set in step S6304 and processed in step S6305, is attached to thedocument image comprehension data 5111.

Hereinafter, each of the above-described processing is described indetail.

FIGS. 44 to 47 are flowcharts showing document image comprehension datastorage processing in step S6305 in FIG. 43, executed in accordance withthe set storage level.

Referring to FIG. 44, the storage level set in step S6304 is determinedin step S6501, and document image comprehension data storage processingaccording to respective storage levels are performed (step S6502, S6503,S6504).

When the determined storage level is “level 1”, the processing shown inthe flowchart in FIG. 45 is executed in step S6502.

In step S6601 in FIG. 45, the layout analysis data of each area isanalyzed to obtain the area attribute. In step S6602, if the areaattribute is “texts”, a binary image is selected in step S6603 to beextracted. Then in step S6605, the partial image is extracted by usingcoordinate data (X, Y), width, and height of the area represented by thelayout analysis data 5107.

If the area attribute is not “texts” in step S6602, a multivalued imagewhich is the original image data 5103 is selected in step S6604. Then instep S6605, the partial image is extracted by using coordinate data,width, and height of the area represented by the layout analysis data5107.

In step S6606, the type of extracted partial image is determined. If itis a binary image type, compression processing for binary images (e.g.,MMR or the like) is performed in step S6607. The compressed partialimage is stored as document image comprehension data 5111 (FIG. 48) instep S6609. At this step, by embedding information indicative of thecompression method in the document image comprehension data 5111, thecompressed image can be decompressed by the document image comprehensiondata reproduction unit 5114.

Meanwhile, if the extracted partial image is a multivalued image type instep S6606, compression processing for multivalued images (e.g., JPEG orthe like) is performed in step S6608. The compressed partial image isstored as document image comprehension data (FIG. 48) in step S6609.

FIG. 48 shows a data structure of the document image comprehension data5111 according to the thirteenth embodiment.

As exemplified in FIG. 48, partial image data comprises: an ID uniquelyassigned to the area, the width of the extracted area, height of theextracted area, type of image, compression method employed, image size,and compressed extracted image. Herein, the assigned ID corresponds tothe sequence of area extraction performed by the layout analyzing unit.

FIG. 49A is a table showing identification numbers for the type ofimage, and FIG. 49B is a table showing identification numbers forcompression methods.

Referring back to FIG. 45, the control proceeds to step S6610 fordetermining whether or not there is a remaining area to be processed. Ifthere is a remaining area, the control returns to step S6601 forrepeating the above-described processing with respect to the remainingarea, whereas if there is no remaining area in step S6610, the documentimage comprehension data storage processing ends.

When the storage level determined in step S6501 in FIG. 44 is “level 2”,storage processing according to the flowchart in FIG. 46 is performed.

In step S6701, the entire image is stored as image data. At this step,according to the thirteenth embodiment, since the entire image inputtedin step S6301 is multivalued image data, the data is stored with acompression method for multivalued images (e.g., JPEG or the like). Aspecific example of document image comprehension data in the case of theentire image data is shown in FIG. 48. More specifically, the entireimage data includes: the width, height, type of image, compressionmethod, image size, and compressed image.

In step S6702, the layout analysis data 5107 of each area is analyzed toobtain the area attribute. In step S6703, if the area attribute is“texts”, the binary image 5105 is selected in step S6704 as an entireimage to be extracted. Then in step S6705, the partial image isextracted by using coordinate data, width, and height of the arearepresented by the layout analysis data 5107. In step S6706, compressionprocessing for binary images (e.g., MMR or the like) is performed, andthe compressed partial image is stored as document image comprehensiondata 5111 (FIG. 48).

If the area attribute is not “texts” in step S6703, the control proceedsto step S6707 and storage processing of the partial image is notperformed with respect to this area. Subsequent to steps S6706 or S6703,the control proceeds to step S6707 for determining whether or not thereis a remaining area to be processed. If there is a remaining area, thecontrol returns to step S6702 for repeating the above-describedprocessing with respect to the remaining area, whereas if there is noremaining area, the storage processing of the document imagecomprehension data 5111 ends.

When the storage level determined in step S6501 in FIG. 44 is “level 3”,storage processing according to the flowchart in FIG. 47 is performed.

In step S6801 in FIG. 47, the entire image is stored as image data. Instep S6802, the layout analysis data 5107 of each area is analyzed toobtain the area attribute. In step S6803, if the area attribute is“texts”, the binary image 5105 is selected in step S6804 as the entireimage to be extracted. Then in step S6805, the partial image isextracted by using coordinate data, the width, and height of the arearepresented by the layout analysis data 5107.

If the area attribute is not “texts” in step S6803, a multivalued imagewhich is the original image data 5103 is selected in step S6806 as theentire image to be extracted. Then in step S6805, the partial image isextracted by using the coordinate data, width, and height of the arearepresented by the layout analysis data 5107.

In step S6807, the type of extracted partial image is determined. If itis a binary image type, compression processing for binary images (e.g.,MMR or the like) is performed in step S6808. The compressed partialimage is stored as document image comprehension data (FIG. 48) in stepS6809. At this step, by embedding information indicative of thecompression method in the document image comprehension data, thecompressed image can be decompressed in the document image comprehensiondata reproduction processing.

Meanwhile, if the extracted partial image is a multivalued image type instep S6807, compression processing for multivalued images (e.g., JPEG orthe like) is performed in step S6810. The compressed partial image isstored as document image comprehension data (FIG. 48) in step S6809.Then in step S6811, it is determined whether or not there is a remainingarea to be processed. If there is a remaining area, the control returnsto step S6802 for repeating the above-described processing with respectto the remaining area, whereas if there is no remaining area in stepS6811, the document image comprehension data storage processing ends.

Further, according to the thirteenth embodiment, the document imagecomprehension data 5111, generated and stored by the document imagecomprehension data storage 5109, is outputted to the network 5212 andtransmitted to users through the network 5212.

In the user side that receives the data, a reproduced image is outputtedaccording to the processing shown in the flowchart in FIG. 50.

In step S6401 in FIG. 50, document image comprehension data 5111 isinputted by the document image comprehension data input unit 5112. Instep S6402, the reproduction level setting unit 5113 sets thereproduction level of the document image. According to the thirteenthembodiment, an operator can set any of the three parameters, “level 1”to “level 3”. The operator's reproduction-level setting is performed byinput means, e.g., a keyboard or touch panel or the like, of theterminal in the system shown in FIG. 2 as similar to step S6304.However, in the case of reproduction-level setting, a selectablereproduction level is limited depending on the storage level at whichthe inputted document image comprehension data is stored. Thus, aselectable reproduction level is determined based on the storage levelindicated by the tag attached to the document image comprehension data5111, and the obtained reproduction level may be displayed on a displayscreen to be presented to the operator. Then in step S6403, based on thelayout analysis data 5107 and reproduction level set in step S6402, thedocument image comprehension data reproduction unit 5114 generatesreproduced document image data 5115 from the document imagecomprehension data 5111 while adaptively changing a reproductioncondition.

FIG. 51 is a flowchart showing image reproduction processing based onthe document image comprehension data 5111, which is performed in stepS6403 in FIG. 50.

In step S7001 in FIG. 51, it is determined whether or not the entireimage data is included in the document image comprehension data 5111. Ifthe entire image data is not included, the control proceeds to stepS7003 where document image comprehension data reproduction processing isforcefully performed in “level 1”. If the entire image data is included,the control proceeds to step S7002 where the reproduction level set instep S6402 is determined. Then, the control branches to reproductionprocessing (S7003 to S7005) of the document image comprehension dataaccording to respective levels.

FIG. 52 is a flowchart showing the reproduction processing (S7003) inlevel 1.

In step S7101 in FIG. 52, a white background image is generated to beused as a background of the reproduced document image. In step S7102,the partial image data is extracted from the document imagecomprehension data 5111. Based on the extracted partial image data andcoordinate data thereof, in step S7103, the partial image is synthesizedwith the white background image, thereby reproducing the image.

An example of reproduction image synthesizing processing in step S7103is described with reference to the flowchart in FIG. 55.

In step S7401 in FIG. 55, the type of partial image is extracted fromthe document image comprehension data 5111. If it is determined in stepS7402 that the type of image is the “binary image type”, pseudo 24-bitconversion is performed in step S7403 by respectively converting blackand white pixels of the binary image to black and white pixels of a24-bit multivalued image.

In the thirteenth embodiment, assume that a black pixel of the binaryimage is expressed by 1, and a white pixel of the binary image isexpressed by 0. A black pixel of the 24-bit multivalued image isexpressed by R=0, G=0, B=0, and a white pixel of the 24-bit multivaluedimage is expressed by R=255, G=255, B=255 (R: red component; G: greencomponent; B: blue component, each having 8-bit value).

In step S7402, if it is determined that the type of image is the “24-bitmultivalued image type”, the partial image without being processed isused for synthesizing processing. In step S7404, logical operation isperformed on each pixel of the partial image with respect to thebackground image and partial image so as to generate a synthesizedimage.

In the thirteenth embodiment, logical operation is performed such that awhite pixel (R=255, G=255, B=255) of the background image, which issynthesized with a black pixel of the partial image (R=0, G=0, B=0),forms a black pixel (R=0, G=0, B=0).

When the reproduction image synthesizing processing (step S7103 in FIG.52) for one partial image is completed in the foregoing manner, whetheror not there is a remaining area is determined in step S7104. If thereis a remaining area, the control returns to step S7102 for repeating theabove-described processing with respect to the remaining area, whereasif there is no remaining area, the document image comprehension datareproduction processing ends.

Referring back to FIG. 50, after reproduction processing is performed inthe above-described manner, the reproduced image is outputted in stepS6404 by the monochrome printer 208, 211 or the like serving as theimage output unit 5116.

Meanwhile, in step S7002 in FIG. 51, if it is determined that thereproduction level is “level 2”, the control proceeds to step S7004 forperforming document image comprehension data reproduction processingaccording to the flowchart in FIG. 53.

In step S7201 in FIG. 53, as similar to the foregoing “reproductionlevel 1”, a white background image is generated to be used as abackground of the reproduced document image. In step S7202, the partialimage data is extracted from the document image comprehension data 5111.In step S7203, the layout analysis data 5107 of each area is analyzed toobtain an area attribute for each area. If the area attribute is“texts”, the partial image is synthesized with the white backgroundimage in step S7205 based on the extracted partial image data andcoordinate data thereof, thereby reproducing the image. If the areaattribute is not “texts”, the control proceeds to step S7204 where thepartial image is extracted from the entire image, by using thecoordinate data. Then in step S7205, the partial image is synthesizedwith the white background image, thereby reproducing the image.

When the reproduction image synthesizing processing (step S7205 in FIG.53) for one partial image is completed in the foregoing manner, whetheror not there is a remaining area is determined in step S7206. If thereis a remaining area, the control returns to step S7202 for repeating theabove-described processing with respect to the remaining area, whereasif there is no remaining area, the document image comprehension datareproduction processing ends.

Referring back to FIG. 50, after reproduction processing is performed inthe above-described manner, the reproduced image is outputted in stepS6404 by a monochrome printer serving as the image output unit 5116.

Meanwhile, in step S7002 in FIG. 51, if it is determined that thereproduction level is “level 3”, the control proceeds to step S7005 forperforming document image comprehension data reproduction processingaccording to the flowchart in FIG. 54.

In step S7301 in FIG. 54, the entire image data is extracted to be usedas a background of the reproduced document image. In step S7302, partialimage data is extracted from the document image comprehension data 5111.In step S7303, the layout analysis data 5107 of each area is analyzed toobtain an area attribute for each area. If the area attribute is“texts”, the partial image is synthesized with the background image instep S7304 based on the extracted partial image data and coordinate datathereof, thereby reproducing the image. If the area attribute is not“texts”, the control proceeds to step S7305 and synthesizing processingof the partial image is not performed with respect to this area.

When the reproduction image synthesizing processing (step S7304 in FIG.54) for one partial image is completed in the foregoing manner, whetheror not there is a remaining area is determined in step S7305. If thereis a remaining area, the control returns to step S7302 for repeating theabove-described processing with respect to the remaining area, whereasif there is no remaining area, the document image comprehension datareproduction processing ends.

Referring back to FIG. 50, after reproduction processing is performed inthe above-described manner, the reproduced image is outputted in stepS6404 by a monochrome printer serving as the image output unit 5116.

As has been set forth above, according to the thirteenth embodiment,since an inputted document image is divided into partial images, storedand reproduced in accordance with image attributes of the documentimage, the amount of data of the stored document image can be reduced,and high quality can be achieved in reproduced images.

Fourteenth Embodiment

Next, the fourteenth embodiment of the present invention is described.In the above-describe thirteenth embodiment, the storage-level settingprocessing (S6304 in FIG. 43) or reproduction-level setting processing(S6402 in FIG. 50) are performed based on setting operation of anoperator. However, according to the fourteenth embodiment, theseprocessings are automatically performed.

Hereinafter, processing steps of the automatic storage-level settingprocessing are described with reference to the flowchart in FIG. 56.

According to the fourteenth embodiment, when the storage-level settingprocessing (S6304) is started, the layout analysis data 5107 of eacharea is extracted in step S6901. Next in step S6902, the size ofnon-text area is inspected.

In the fourteenth embodiment, in a case where the height (h) of the areais larger than a threshold value (Th), or where the width (w) of thearea is larger than a threshold value (Tw), or where the area (s) islarger than a threshold value (Ts), the inspection result is determinedas “NG”, otherwise, the inspection result is determined as “OK”. Whenthe inspection result in step S6902 is “OK”, the control proceeds tostep S6903 where “storage level 1” is set. If the inspection result instep S6902 is “NG”, the control proceeds to step S6904 for executinganother inspection. In step S6904, the number of non-text areas isinspected.

In step S6904, in a case where the number (n) of the non-text areas islarger than a threshold value (Tn), the inspection result is determinedas “NG”, otherwise, the inspection result is determined as “OK”. Whenthe inspection result in step S6904 is “OK”, “storage level 1” is set instep S6903. If the inspection result in step S6904 is “NG”, inspectionin step S6905 is executed. In step S6905, the level of overlaps betweena text area and non-text area is inspected.

In step S6905, in a case where an overlapping area (d) between the textarea and non-text area is larger than a threshold value (Td), theinspection result is determined as “NG”, otherwise, the inspectionresult is determined as “OK”. When the inspection result in step S6905is “OK”, the control proceeds to step S6906 where “storage level 2” isset. If the inspection result in step S6905 is “NG”, the controlproceeds to step S6907 where “storage level 3” is set, and the automaticstorage-level setting processing ends.

The present invention can be applied to a system constituted by aplurality of devices (e.g., host computer, interface, reader, printer)or to an apparatus comprising a single device (e.g., copying machine,facsimile machine).

Further, the objects of the present invention can also be achieved byproviding a storage medium (or recording medium), storing program codesof software realizing the functions according to the first to fourteenthembodiments, to a computer system or apparatus, reading the programcodes, by a CPU or MPU of the computer system or apparatus, from thestorage medium, then executing the program. In this case, the programcodes read from the storage medium realize the functions according tothe above-described embodiments, and the storage medium storing theprogram codes constitutes the present invention. Furthermore, besidesaforesaid functions according to the above embodiments are realized byexecuting the program codes which are read by a computer, the presentinvention includes a case where an OS (operating system) or the likeworking on the computer performs a part or the entire processes inaccordance with designations of the program codes and realizes functionsaccording to the above embodiments.

Furthermore, the present invention also includes a case where, after theprogram codes read from the storage medium are written in a functionexpansion card which is inserted into the computer or in a memoryprovided in a function expansion unit which is connected to thecomputer, CPU or the like contained in the function expansion card orunit performs a part or the entire process in accordance withdesignations of the program codes and realizes functions of the aboveembodiments.

As has been set forth above, according to the thirteenth and fourteenthembodiments, the following effects are attained:

-   (1) The amount of data is reduced when a document image is stored;-   (2) The load imposed on network traffic is reduced when a document    image is transmitted;-   (3) High quality of a document image suitable for reuse can be    maintained when storing or transmitting the document image;-   (4) Image quality deterioration or data omission can be prevented    when outputting a document image; and-   (5) Electronic use of documents, such as desktop publishing (DTP),    is facilitated.

In addition, the amount of data of an inputted document image can bereduced when storing the data, and the stored data can be read andreproduced with high quality.

The present invention is not limited to the above embodiments andvarious changes and modifications can be made within the spirit andscope of the present invention. Therefore, to apprise the public of thescope of the present invention, the following claims are made.

1. An image processing apparatus comprising: an input unit configured toinput multi-valued image data of a document; a binary image generationunit configured to generate binary image data from the inputtedmulti-valued image data; a layout analysis unit configured to divide thegenerated binary image data into areas for each attribute, and generatelayout information of the divided areas; a first partial imageextraction unit configured to extract, from the binary image data, apartial image having text-attribute on the basis of the layoutinformation; a second partial image extraction unit configured toextract, from the multi-valued image data, a partial image havingnon-text-attribute on the basis of the layout information; an encryptionunit configured to encrypt one of partial images extracted by said firstand second partial image extraction unit, as a partial image to beencrypted; and an storage unit configured to store the encrypted datagenerated by said encryption unit, the partial image which is not to beencrypted and has been extracted by either said first or second partialimage extraction units, and the layout information.
 2. The apparatusaccording to claim 1, wherein the area to be encrypted is an area havingtext-attribute, and the area not to be encrypted is an area havingnon-text-attribute.
 3. The apparatus according to claim 1, furthercomprising an encryption target attribute setting unit configured to setan attribute to be encrypted on the basis of an operator's instruction,wherein the area to be encrypted is an area having the attribute set bysaid encryption target attribute setting unit, and the area not to beencrypted is an area having an attribute other than the attribute set bysaid encryption target attribute setting unit.
 4. The apparatusaccording to claim 1, further comprising an encryption target settingunit configured to display areas for each attribute in accordance withthe layout information and set an area, as an attribute to be encrypted,in accordance with an operator's instruction, wherein the area to beencrypted is an area set by said encryption target setting unit, and thearea not to be encrypted is an area other than the area set by saidencryption target setting unit.
 5. The apparatus according to claim 1,further comprising an output unit configured to output the encrypteddata, the partial image and the layout information stored by saidstorage unit to another apparatus.
 6. An image processing apparatuscomprising: an input unit configured to input multi-valued image data ofa document; a binary image generation unit configured to generate binaryimage data from the inputted multi-valued image data; a layout analysisunit configured to divide the generated binary image into areas for eachof attributes, and generate layout information of the divided areas; afirst partial image extraction unit configured to extract, from thebinary image, a partial image having text-attribute in accordance withthe layout information; a character recognition unit configured toperform character recognition with respect to the partial image havingtext-attribute extracted by said first partial image extraction unit; asecond partial image extraction unit configured to extract, from themulti-valued image data, a partial image having non-text attribute inaccordance with the layout information; an encryption unit configuredto, if an area to be encrypted is the area having text-attribute,generate encrypted data by encrypting character recognition result bysaid character recognition unit; an storage unit configured to store theencrypted data generated by said encryption unit, the partial imageextracted by said second partial image extraction unit, and the layoutinformation.
 7. The apparatus according to claim 6, wherein saidencryption unit, if an area to be encrypted is the area havingtext-attribute, encrypts both of character recognition result by saidcharacter recognition unit and the partial image, which has beenperformed character recognition, extracted by said first partial imageextraction unit.
 8. The apparatus according to claim 6, furthercomprising an output unit configured to output the encrypted data, thepartial image and the layout information stored by said storage unit toanother apparatus.
 9. An image processing method comprising the stepsof: inputting multi-valued image data of a document; generating binaryimage data from the inputted multi-valued image data; dividing thegenerated binary image data into areas for each attribute, andgenerating layout information of the divided areas; a first extractionstep of extracting, from the binary image data, a partial image havingtext-attribute on the basis of the layout information; a secondextraction step of extracting, from the multi-valued image data, apartial image having non-text-attribute on the basis of the layoutinformation; encrypting one of the partial images extracted in saidfirst or second partial image extraction step, as a partial image to beencrypted; and storing the encrypted data generated in said encryptionstep, the partial image which is not encrypted and has been extracted ineither said first or second partial image extraction step, and thelayout information.
 10. The method according to claim 9, wherein thearea to be encrypted is an area having text-attribute, and the area notto be encrypted is an area having non-text-attribute.
 11. The methodaccording to claim 9, further comprising setting an attribute to beencrypted on the basis of an operator's instruction, wherein the area tobe encrypted is an area having the attribute set in said setting step,and the area not to be encrypted is an area having an attribute otherthan the attribute set in said setting step.
 12. The method according toclaim 9, further comprising displaying areas for each attribute inaccordance with the layout information and setting an area, as anattribute to be encrypted, in accordance with an operator's instruction,wherein the area to be encrypted is an area set in said setting step,and the area not to be encrypted is an area other than the area set insaid setting step.
 13. The method according to claim 9, furthercomprising outputting the encrypted data, the partial image and thelayout information stored in said storing step to another apparatus. 14.An image processing method comprising the steps of: inputtingmulti-valued image data of a document; generating binary image data fromthe inputted multi-valued image data; dividing the generated binaryimage into areas for each of attributes, and generating layoutinformation of the divided areas; a first extraction step of extracting,from the binary image, a partial image having text-attribute inaccordance with the layout information; performing character recognitionwith respect to the partial image having text-attribute extracted insaid first extraction step; a second extraction step of extracting, fromthe multi-valued image data, a partial image having non-text attributein accordance with the layout information; if an area to be encrypted isthe area having text-attribute, encrypting a character recognitionresult of said character recognition step; storing the encrypted datagenerated in said encrypting step, the partial image extracted in saidsecond extraction step, and the layout information.
 15. The methodaccording to claim 14, wherein said encrypting step, if an area to beencrypted is the area having text-attribute, encrypts both of acharacter recognition result of said character recognition and thepartial image, which has undergone character recognition, extracted insaid first extraction step.
 16. The method according to claim 14,further comprising outputting the encrypted data, the partial image andthe layout information stored in said storing step to another apparatus.17. A computer-readable medium embodying a program for causing anapparatus to execute an image processing method that comprises the stepsof: inputting multi-valued image data of a document; generating binaryimage data from the inputted multi-valued image data; dividing thegenerated binary image data into areas for each attribute, andgenerating layout information of the divided areas; a first extractionstep of extracting, from the binary image data, a partial image havingtext-attribute on the basis of the layout information; a secondextraction step of extracting, from the multi-valued image data, apartial image having non-text-attribute on the basis of the layoutinformation; encrypting one of the partial images extracted in saidfirst or second partial image extraction step, as a partial image to beencrypted; and storing the encrypted data generated in said encryptionstep, the partial image which is not encrypted and has been extracted ineither said first or second partial image extraction step, and thelayout information.
 18. The computer readable medium according to claim17, wherein the area to be encrypted is an area having text-attribute,and the area not to be encrypted is an area having non-text-attribute.19. The computer readable medium according to claim 17, furthercomprising setting an attribute to be encrypted on the basis of anoperator's instruction, wherein the area to be encrypted is an areahaving the attribute set in said setting step, and the area not to beencrypted is an area having an attribute other than the attribute set insaid setting step.
 20. The computer readable medium according to claim17, further comprising displaying areas for each attribute in accordancewith the layout information and setting an area, as an attribute to beencrypted, in accordance with an operator's instruction, wherein thearea to be encrypted is an area set in said setting step, and the areanot to be encrypted is an area other than the area set in said settingstep.
 21. The computer readable medium according to claim 17, furthercomprising outputting the encrypted data, the partial image and thelayout information stored in said storing step to another apparatus. 22.A computer-readable medium embodying a program for causing an apparatusto execute an image processing method that comprises the steps of:inputting multi-valued image data of a document; generating binary imagedata from the inputted multi-valued image data; dividing the generatedbinary image into areas for each of attributes, and generating layoutinformation of the divided areas; a first extraction step of extracting,from the binary image, a partial image having text-attribute inaccordance with the layout information; performing character recognitionwith respect to the partial image having text-attribute extracted insaid first extraction step; a second extraction step of extracting, fromthe multi-valued image data, a partial image having non-text attributein accordance with the layout information; if an area to be encrypted isthe area having text-attribute, encrypting a character recognitionresult of said character recognition step; storing the encrypted datagenerated in said encrypting step, the partial image extracted in saidsecond extraction step, and the layout information.
 23. The computerreadable medium according to claim 22, wherein said encrypting step, ifan area to be encrypted is the area having text-attribute, encrypts bothof a character recognition result of said character recognition and thepartial image, which has undergone character recognition, extracted insaid first extraction step.
 24. The computer readable medium accordingto claim 22, further comprising outputting the encrypted data, thepartial image and the layout information stored in said storing step toanother apparatus.